An Automated Testing Environment to support Operational Profiles of Software Intensive Systems

Raytheon Systems Company is a defense electronics company that has been actively engaged in a software process improvement effort at the organizational and program levels for over a decade. The company uses the Software Engineering Institute’s (SEI) Capability Maturity Model (CMM) as a basis for their improvement efforts and has been formally assessed at level 3 (defined). The project is a real-time embedded software application using a heterogeneous computer architecture consisting of the two fundamental computing environments; • PowerPC single board computer (SBC) using Ada for embedded command and control, • Digital signal processors (DSP) using the C programming language for a primarily signal processing algorithm based application. There are significant real-time constraints within the signal processing application. The program follows a tailored version of DoD-STD-2167A and MIL-STD-498 documentation standards. The project is built upon an Integrated Product Team (IPT) structure. There is a fully defined software development process as well as system engineering and hardware development processes. Our testing approach is based on the concepts of software testing based on statistical principles. The statistical testing approach to software treats the software like a statistical experiment. A statistical subset of all possible software uses is first generated. Performance on this subset is used to form conclusions about operational performance based on the usage model developed. The expected operational use is represented in a usage model of the software. Test cases are then randomly generated from the usage model. These tests are executed in an operational environment. Failures are interpreted according to mathematical and statistical models. This paper will focus on the successes and issues associated with developing a statistical testing environment for an industrial software project. The paper will also describe how both statistical testing based on software models and traditional testing based on unit and other functional tests can be combined into an effective approach to testing large software intensive systems.